Joarder Mohammad Mustafa Kamal: PhD Thesis

More than 3.26 billion Internet users are using Web and mobile applications every day for retail services and businesses processing; information management and retrievals; media and social interactions; gaming and entertainments, etc. With the rapid growth in simultaneous users and data volume, such applications often rely on distributed databases to handle large magnitudes of On-Line Transaction Processing (OLTP) requests at the Internet scale. These databases employ user-defined partitioning schemes during their initial application design phase to scale-out concurrent writing of user data into a number of shared-nothing servers. However, processing Distributed Transactions (DTs), that involve data tuples from multiple servers, can severely degrade the performance of these databases. In addition, transactional patterns are often influenced by the frequent data tuples in the database, resulting in significant changes in the workload that none of the static partitioning schemes can handle in real-time. This thesis addresses the abovementioned challenge through workload-aware incremental data repartitioning. The primary goal is to minimally redistribute data tuples within the system with a view to minimising the adverse impact of DT without adversely affecting the data distribution load balance. Furthermore, to support both range and consistent-hash based data partitions used in OLTP systems, a distributed yet scalable data lookup process, inspired by the roaming protocol in mobile networks, is introduced. Moreover, additional challenge lies on how to reduce the computational complexities while processing large volume of transactional workload very quickly to make real-time decisions. These complexities are addressed in three distinct ways hide the workload network complexities during analysis by using different level of abstractions; find the appropriate size of observation window to decide how far to look back in the past for workload analysis; and finally, incrementally (i.e., how frequently) trigger the repartitioning process based on a threshold. Some well-defined Key Performance Indicator (KPI) metrics for database repartitioning are defined and rigorous sensitivity analyses are performed on different types of transaction generation models and observation window sizes. In the first stage, graph-theoretic higher level abstractions such as graphs, hypergraphs, and their compressed forms are used to deliver sub-optimal repartitioning decisions using graph min-cut techniques. In order to deliver optimal decisions, a greedy approach is further developed, which ranks the transactions in terms of the measure of repartitioning objectivity and performs transaction-level optimisation to maximise a particular KPI. Finally, transactional stream mining is used to construct a representative sub-network of frequent transactions and perform transaction-level adaptive repartitioning, which eventually reduces the computational complexity, while concomitantly improving the performance due to filtering out a significant volume of infrequent transactions, that have little bearing on the KPIs. Simulation-based experimental evaluations demonstrate that the proposed workload-aware incremental repartitioning schemes outperform the existing static and dynamic repartitioning approaches to be considered effective for production deployment. Declaration This thesis contains no material which has been accepted for the award of any other degree or diploma at any university or equivalent institution and that, to the best of my knowledge and belief, this thesis contains no material previously published or written by another person, except where due reference is made in the text of the thesis. Signature: Print Name: Joarder Mohammad Mustafa Kamal Date: 21 February 2017 Acknowledgements First and foremost, I would like to express my gratefulness to the Almighty, Allah, the most generous for blessing me with the physical and mental ability, sustenance, and patience to undertake this research successfully. Next, I need to thank my parents and parent-in-laws for continuously encouraging me to purse my dream. Without their ultimate sacrifice, it would not be possible for me to reach this end. I am extremely fortunate to have Prof. Manzur Murshed as my main supervisor. Without his persistent guidance, insightful directions, and support, this research would not have been successfully completed. I am also indebted to him and his family for allowing me the time over a number of weekends to have research meetings at their residence when I moved to Sydney for work. I am also thankful to A/Prof. Joarder Kamruzzaman, Prof. Rajkumar Buyya, and Prof. Medhat Gaber for their valuable advices and guidelines. Additionally, I am grateful to Dr. Gour Karmakar, A/Prof. Madhu Chetty, Prof. Sue McKemmish, and Dr. Mortuza Ali for often enlightening my ideas and findings. I am also thankful to Prof. Donald Kossmann, Dr. Carlo Curino, Dr. Evan Jones, and Dr. Djellel Difallah for their consecutive correspondences, suggestions and comments on my works. I specially thank the Australian Government and Monash University for supporting my research with the IPRS and APA scholarships. I also thank the staffs at the Monash Campus Cluster (MCC), the Faculty and Gippsland School of IT, and Monash University Institute of Graduate Research (MIGR) for all of their administrative supports towards the successful completion of my research. A special thanks to Mrs. Freda Webb for her exceptional cooperation as my student advisor. I would also like to thank Dr. Gillian Fulcher for proofreading this thesis. I am ever thankful to my dear friends and well-wishers Fasahat Siddiqui, Ahammed Shereif, Ahsan Raja Chowdhury, Rumana Nazmul, Hasanul Ferdaus, Ali Naqi, Tori Stratford, Mamun Abdullah, Shahriar Kaisar, and Shakif Azam for their selfless support, encouragements, and motivations that never let me feel alone. Finally, I specially thank my wife, Rushmila, to share this enormous journey with me and sacrifice the most in doing so. At the beginning of my PhD, she gave up her career to join me in Australia and has shared all the difficult moments in all these past years. Without her everlasting love and continual support, I would not be able to submit my thesis today. Dedicated to my parents, and to my wife who sacrificed the most for this research.